Papers with Transformer-based Language Models
Lightweight Transformers for Conversational AI (2022.naacl-industry)
Copied to clipboard
| Challenge: | Commercial dialogue systems typically require a small footprint and fast execution time, but recent trends are in the other direction, resulting in difficulties in model deployment. |
| Approach: | They build Transformer-based Language Models from scratch on large corpora of conversational data and compare their performance against BERT and other strong baselines on dialogue probing tasks. |
| Outcome: | The proposed model outperforms existing models on dialogue probing tasks and can be fine-tuned on a single consumer GPU card. |
Reversed Attention: On The Gradient Descent Of Attention Layers In GPT (2025.naacl-long)
Copied to clipboard
| Challenge: | In this work, we examine the attention maps obtained from the backward pass of attention, which we call "Reversed Attention" (RA). |
| Approach: | They propose to use a method called "attention patching" to alter the forward pass of attention without modifying the model's weights. |
| Outcome: | The proposed method enables the model to alter the forward pass of attention without altering the model’s weights. |
Evaluating Lexical Proficiency in Neural Language Models (2025.acl-long)
Copied to clipboard
| Challenge: | Recent advances in Natural Language Processing have been significantly shaped by the Deep Learning tsunami and the introduction of Transformer-based Language Models. |
| Approach: | They validated a framework to assess the lexical proficiency and linguistic creativity of Transformer-based Language Models (LMs) by analyzing performance of LMs of different sizes across tasks involving the generation, definition, and contextual usage of lexicals, neologisms, and nonce words. |
| Outcome: | The framework evaluates LMs in mono- and multilingual configuration across tasks involving the generation, definition, and contextual usage of lexicalized words, neologisms, and nonce words. |
NeuroPrune: A Neuro-inspired Topological Sparse Training Algorithm for Large Language Models (2024.findings-acl)
Copied to clipboard
Amit Dhurandhar, Tejaswini Pedapati, Ronny Luss, Soham Dan, Aurelie Lozano, Payel Das, Georgios Kollias
| Challenge: | Transformer-based Language Models have become ubiquitous in natural language processing due to impressive performance on various tasks. |
| Approach: | They explore how sparsity affects network topology by exploiting mechanisms seen in biological networks . they show that model-agnostic sparsities are performant across diverse NLP tasks . |
| Outcome: | The proposed model-agnostic sparsity approaches are performant and efficient across NLP tasks. |
Backward Lens: Projecting Language Model Gradients into the Vocabulary Space (2024.emnlp-main)
Copied to clipboard
| Challenge: | Recent interpretability methods project weights and hidden states obtained from the forward pass to the models’ vocabularies, helping to uncover how information flows within LMs. |
| Approach: | They propose to cast a gradient matrix as a low-rank linear combination of forward and backward passes’ inputs and then to project these gradients into vocabulary items. |
| Outcome: | The proposed method can be cast as a low-rank linear combination of forward and backward passes’ inputs and project these gradients into vocabulary items. |
Disentangling Transformer Language Models as Superposed Topic Models (2023.emnlp-main)
Copied to clipboard
| Challenge: | Topic Modelling is an established area of research where the quality of a given topic is measured using coherence metrics. |
| Approach: | They propose a weight-based approach to search and disentangle decoder-only TLM by a Wikipedia corpus. |
| Outcome: | The proposed approach can be applied to GPT-2 models and LLaMA models. |
Language Detoxification with Attribute-Discriminative Latent Space (2023.acl-long)
Copied to clipboard
| Challenge: | Existing methods to detoxify toxic text require excessive memory, computations and time. |
| Approach: | They propose a method to generate toxic text using an attribute-discriminative latent space. |
| Outcome: | The proposed method outperforms baselines on detoxified language and dialogue generation tasks while being time- and memory-efficient. |
ILCiteR: Evidence-grounded Interpretable Local Citation Recommendation (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing approaches for local citation recommendation map or translate a query to citation-worthy research papers. |
| Approach: | They propose a local citation recommendation task that uses latent evidence spans to recommend papers . proposed system retrieves ranked lists of evidence span and recommended paper pairs . |
| Outcome: | The proposed system retrieves ranked lists of evidence span and recommended paper pairs based on evidence from the existing literature. |